---
title: "Examining Senior Seminar and Curricular Reform at an HBCU"
author: "Matthew B. Platt"
date: "March 5, 2020"
output:
  tufte::tufte_handout: default
fontfamily: mathpazo
fontsize: 12pt
geometry: margin=1in
header-includes: \usepackage{dcolumn}
link-citations: yes
bibliography: senior.bib
biblio-style: apsr
---

```{r setup, include=FALSE}

library(tufte)
# invalidate cache when the tufte version changes
knitr::opts_chunk$set(tidy = FALSE, cache.extra = packageVersion('tufte'), message = FALSE, echo = FALSE)
options(htmltools.dir.version = FALSE)

library(tidyverse)
library(readxl)

load("Data/AnalysisData/analysis.RData")

```

# Purpose

Since its publication in 1991, the Wahlke Report has been a focal document among political scientists who care about curricular design. However, the Wahlke Task Force does not get enough credit for recognizing the fundamental connection between curricular design and Dungeons and Dragons (DnD).  A great DnD campaign is about providing the player characters (PCs) with an escalating set of engaging encounters that require them to develop new skills, abilities, strategies, and tactics in order to progress to higher levels. At the end of the campaign the PCs are tested by a boss fight that requires them to employ all that they've learned in order to survive. In other words, it is a program of sequential learning that culminates in a senior capstone in which students "integrate knowledge from the totality of their program" [@Wahlke1991, 57]. As with DnD, students should have had fun, maybe made some friends, learned a lot, and appreciated the challenge of their "boss fight" senior project.

What happens when our students keep losing the boss fights? In a game context, the problem is likely that the players were not equipped with the right weapons or items; perhaps they skipped some encounters so their powers and abilities are not at a high enough level; or maybe the encounters did not require the players to use the kinds of strategies and tactics that would be necessary for that final battle. This paper uses the methodology developed by @Siver2016 to investigate these possibilities at an all-male, all-black liberal arts college in the south. Although the demographics of this study are atypical, the lessons are broadly applicable: faculty need a shared vision of curricular outcomes; that shared vision can and should be closely aligned with the mission of the department and college; and curriculum is more about student experiences than faculty intentions.  

The paper proceeds in four sections. Section 1 briefly provides context about the department's curriculum and the reforms that this study was originally intended to assess. Section 2 provides a basic description of the data that will be used throughout this examination. Next, I present the key finding of this investigation: student performance in the capstone course is driven almost entirely by the section of Senior Seminar in which they enroll. The last section discusses broader questions about how to achieve uniform standards for the capstone while maintaining methodological pluralism.

# The Reformed Curriculum

In Spring 2016 the faculty were dissatisfied with the relatively low quality of senior capstone projects. Students did not seem prepared to do original research, personnel constraints often resulted in a mismatch between faculty expertise and students' research interests, and a significant minority of students were completing the capstone prior to taking the research methods course.  We made four changes to the curriculum to address these problems. First, the track system, which required students to take three upper-division electives in one subfield, was eliminated. The idea was to provide students with an opportunity to explore their interests and different methodological approaches. This also created flexibility in how personnel could be allocated to courses. Second, we offered multiple sections of Senior Seminar (the capstone course) organized around broad topics like racial capitalism, democracy, and representation. The idea was provide a stronger match between faculty expertise and student interests. Third, a required cognate writing course was replaced with a fourth upper division elective. Fourth, the research methods course was taught as an applied quantitative methods course (rather than a research design course), and the methods course was made a prerequisite for the capstone.

After these reforms, the curriculum was more in line with the sequential learning advocated for in @Wahlke1991. The department's intended path to the capstone is that students will have completed the four subfield introductory courses to establish a baseline as sophomores. The research methods course arms them with the tools to begin producing political science. We want juniors to develop specialized knowledge and interest, apply their new methodological skills, and learn how to engage with the literature through the four upper division electives. Ideally, seniors would emerge from that curriculum with some defined interests, and then they will enroll in the section of senior seminar that most closely matches those research interests. 

# Data Collection

This study examines senior seminar from Spring semester 2016 through Spring semester 2019. Following @Siver2016, there are three components to the study: first, transcripts are used to identify the paths that students actually took towards the capstone; second, the course syllabi for the political science electives are analyzed for whether they could conceivably prepare a student to complete his capstone research; and third, the senior seminar papers are assessed using a common rubric. I adapt the rubric provided by @Siver2016: does the thesis explicitly define and measure its main concepts, does the thesis have a minimum of seven academic sources, is there a critical assessment of the literature, is there a hypothesis that states an explicit relationship between a dependent and independent variable, does the thesis analyze primary data or document, and does the thesis contain some original presentation of data analysis. Each thesis was coded as either having these individual elements or not.

Part of our college's mission is "teaching the history and culture of black people", so I included an additional category -- can the research be classified as "black political science"? The term "Black political science" is loosely defined based on various writings by @Jones2015. For the purposes of this study, capstone research is classified as black political science if 1) black people and their conditions are the central focus of inquiry or 2) the phenomenon being studied is viewed primarily in terms of how it impacts black people. For example, research on black political participation would satisfy the first criterion, and a project on mass incarceration could satisfy the second criterion. 

Table 1 provides an overview of the data. There is transcript data for all 121 students^[Students are counted as a new observation for each time they took the course.] who were enrolled in Senior Seminar. However, twenty-two students never submitted a research paper, two students submitted papers that were plagiarized (and are thus not included in the assessment), and seventeen submitted papers were not made available for this study.^[All seventeen of the papers that were not made available for study come from the Comparative Politics sections of senior seminar.] 

```{r overview}

over <- thesisdata %>% 
  group_by(semfield) %>%
  summarise(count = n(),
            submits = sum(submitted, na.rm=T),
            grades = sum(graded, na.rm=T))

totstudent <- sum(over$count)

total <- c("Total", sum(over$count), sum(over$submits), sum(over$grades))
over <- rbind(over, total)

knitr::kable(over, col.names = c("Field", "Students", "Theses Submitted", "Theses Assessed"), align = 'lccc', padding = 2,
             caption = 'The Number of Senior Theses by Seminar Field, Submission, and Assessment')

```

With this background and set of caveats, we can now explore how students traverse through the curriculum, how well the actual courses map onto our idyllic vision of the curriculum, and how those two factors shape the quality of research papers produced in our Senior Seminar.

# Results

## The Path to the Capstone

I am interested in whether students take methods prior to the capstone, how many upper-level electives are taken prior to the capstone, and whether their capstone research matches their choice of electives. In order to answer these questions, I examined the transcripts of the  `r totstudent` students who were enrolled in Senior Seminar during the period of study.

```{r prereqs}

path <- thesisdata %>%
  mutate(allelect = as.numeric(electives == 4)) %>%
  summarise(scopeprop = round(mean(scope, na.rm=T), digits = 3)*100,
            electprop = round(mean(allelect, na.rm=T), digits = 3)*100,
            preprop = round(mean(prereq, na.rm=T), digits = 3)*100,
            match = round(mean(topicmatch, na.rm = T), digits = 3)*100)

total <- c(nrow(thesisdata), nrow(thesisdata), nrow(thesisdata),  sum(thesisdata$graded, na.rm=T))
path <- rbind(path, total)
path <- cbind(c("Proportion","N"), path)

knitr::kable(path, col.names = c("","Methods", "Electives", "Both", "Topic Match"), align = 'lcccc', padding = 2,
             caption = 'Table 2: Most students take methods but not their electives prior to the capstone.')

# making data for the footnote on lowering threshold to 3 electives instead of 4
path2 <- thesisdata %>%
  mutate(allelect = as.numeric(electives >= 3),
         prereq2 = as.numeric(electives >=3&scope == 1)) %>%
  summarise(scopeprop = round(mean(scope, na.rm=T), digits = 3),
            electprop = round(mean(allelect, na.rm=T), digits = 3),
            preprop = round(mean(prereq2, na.rm=T), digits = 3),
            match = round(mean(topicmatch, na.rm = T), digits = 3))

```

Based on the transcript data, Table 2 shows the proportion of students who have completed the methods and electives requirements prior to taking Senior Seminar. The message from the table is that `r round(path$scopeprop[1], digits=1)`% of students have passed our "Scope and Methods" course prior to taking Senior Seminar. However, only `r round(path$electprop[1], digits=1)`% have completed all four of their upper-level elective courses before taking Senior Seminar. Overall, this means that only `r round(path$preprop[1], digits=1)`% of students enter the capstone "fully prepared" according to our ideal curricular path.^[If we lower the threshold to completing three upper-level electives instead of four, then the numbers rise to `r round(path2$electprop[1], digits=1)`% for electives and `r round(path2$preprop[1], digits=1)`% overall.] Additionally, a minority of students' (`r round(path$match[1], digits=1)`%) capstone fields match the elective courses they completed.^[This number would perhaps change if the seventeen missing theses in comparative politics were incorporated.] Students are not following the prescribed path.

## Are We Teaching Them?

Table 3 summarizes the thirty-seven different courses that were used to satisfy the upper-level elective requirement over the period of study.^[The number of courses is 37 because one of the courses is "none" indicating that the student had not completed the requirement, and one of the courses is "transfer" for students who transferred in a course to satisfy the requirement.] 

```{r elective}

electable <- group_by(electivestack, field) %>%
  summarise(count = n(),
            students = sum(Freq, na.rm=T),
            available = sum(syllabus, na.rm=T),
            resprop = sum(research, na.rm=T)
            )

total <- c("Total", sum(electable$count), sum(electable$students), sum(electable$available), sum(electable$resprop))

levels(electable$field) <- c("American", "Comparative", "English", "IR", "Law", "none", "Theory", "transfer", "Total")

electable <- rbind(electable, total)

knitr::kable(electable, col.names = c("Field", "Courses", "Students", "Syllabus", "Research"), align = 'lcccc', padding = 2,
             caption = 'Table 3: Students gravitate towards law and comparative electives.')


```

The most important takeaway from Table 3 is that only ten of the seventeen courses with an available syllabus include an assignment that is either a full research paper or some major component of the research process (annotated bibliography, literature review, research design, data collection, etc.).^[Another limitation of this analysis is that most of the course syllabi were not available. There are three reasons for the lack of data. First, Morehouse is part of a consortium of colleges that allows students to take courses on any of the three undergraduate campuses. Those courses count for ten of the missing syllabi. Second, there are seven courses that were offered by faculty who are no longer with the college or department. Third, there are two courses that were last taught by current Morehouse political science faculty.]  Table 4 underlines this point by looking at the top ten electives by student enrollment. 

```{r courses}

rank <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
cnum <- c("PSC 351", "ENG 265", "PSC 302", "PSC 385", "PSC 477", "PSC 464", "PSC 350", "PSC 463", "PSC 486", "PSC 322")
ctitle <- c("Intro to Moot Court", "Advanced Composition", "Third World Politics", "Theories of IR", "African Politics",
            "Black Political Thought", "Race and Law", "Contemporary Theories of Justice", "Policy Ideologies", "American Congress")
cstudents <- c(49, 44, 42, 34, 32, 28, 25, 20, 20, 18)
syllabus <- c("no", NA, "no", "no", "yes", "yes", "no", NA, NA, "yes")

electrank <- cbind(rank, cnum, ctitle, cstudents, syllabus)

knitr::kable(electrank, col.names = c("Rank", "Course", "Title", "Students", "Research"), align = 'lcccc', padding = 2,
             caption = 'Table 4: The upper-level electives are not developing research skills')



```

Only three of these ten courses have an explicit research component. Combining these findings with what we discovered about students' paths towards the capstone, the electives are not fully serving the functions for which they are intended. The next section will turn towards whether that matters for the quality of seniors' research projects.

## Does it Matter?

Students are not necessarily following the prescribed path towards the senior capstone, and the path itself may not prepare students in the way it was intended. In this section we explore variation in the quality of students' research projects. Recall, that quality has been operationalized as the six criteria highlighted in @Siver2016 and whether the research can be classified as black political science. We will analyze both the individual components and a score that simply sums across the seven categories. Figure 1 presents a histogram of those summed scores.

```{r histogram, fig.cap = "Figure 1: Most senior projects have three or fewer key attributes.", message=FALSE}

cleanthesis <- filter(thesisdata, graded == 1)

p3 <- ggplot(data = cleanthesis, aes(x = score)) + geom_histogram()
p3

```

The bulk of observations contain three or fewer of the key attributes, but there is a substantial minority of students who completed research projects that have five or more of the attributes of a quality research project. 

```{r attributes}

quality <- thesisdata %>% filter(graded == 1) %>%
  summarise(count = n(),
            sourceprop = mean(sourcecount)*100,
            conceptprop = mean(concepts)*100,
            litprop = mean(litcritique)*100,
            hypoprop = mean(hypothesis)*100,
            primaryprop = mean(primarydata)*100,
            dataprop = mean(dataviz)*100,
            blackprop = mean(blackpol)*100
            )

knitr::kable(quality, digits = 3, col.names = c("Assessed", "Sources", "Concepts", "Critique", "Hypothesis", "Primary Source", "Visualization",
                                    "Black Polisci"), align = 'c', padding = 2,
             caption = 'Table 5: For four out of the seven key attributes, A majority of student projects demonstrate competence.')

```

Table 5 shows the percentage of projects that satisfy each of the seven criteria. Students had the most difficulty in offering an evaluation or critique of the literature (`r round(quality$litprop,digits=1)`%), and only a sizable minority of students included an original presentation of data within their capstone papers. A majority of the research projects had sufficient sources, defined concepts, used primary sources, and could be classified as black political science. 


```{r method-v-score, fig.margin=FALSE, fig.cap="Figure 2: Taking methods does not matter for the quality of the research project.", cache=TRUE, message=FALSE}

# plot scope vs. thesis
thesisdata1 <- thesisdata
thesisdata1$scopefct <- as.factor(thesisdata1$scope)
levels(thesisdata1$scopefct) <- c("No", "Yes")

meanscore1 <- filter(thesisdata1, is.na(score)==F) %>%
  group_by(scopefct) %>%
  summarise(avgscore = mean(score))
  

p4 <- ggplot(meanscore1, aes(y = avgscore, x = scopefct, fill = scopefct)) +
  geom_col() + 
  labs(x = "Completed Methods", y = "Average # of Attributes", fill = "Methods") +
  theme(legend.position = "none")
p4
```


``` {r elective-v-score, fig.cap = "Figure 3: Taking electives does not matter for the quality of the research project.", cache=TRUE, message=FALSE}

# boxplot thesis v. electives
thesisdata1$electord <- as.ordered(thesisdata1$electives)
thesisdata2 <- filter(thesisdata1, graded == 1)

p5 <- ggplot(thesisdata2, aes(x = electord, y = score, fill = electord)) +
  geom_boxplot() + labs(x = "# of Electives", y = "# of Attributes", fill = "Electives") +
  theme(legend.position = "none")
p5

```

Figures 2 and 3 illustrate that completing the intended prerequisites does not affect the quality of capstone projects. The average attribute score for those who completed the methods course is not significantly different (neither statistically nor substantively) from those who did not take the methods course prior to taking Senior Seminar. The path to the capstone does not matter. What matters is the actual version of the capstone in which a student enrolls. Figure 4 plots the average attribute score for capstone projects by academic year and seminar subfield. On average, research projects in the American Politics senior seminar meet more of our criteria than those in the other subfields. 

```{r semfield, fig.cap = "Figure 5: Which Senior Seminar students take matters.", message=FALSE, fig.height=6,fig.width=6}

sumthesis1 <- filter(sumthesis1, grades > 0)

# plot year, score, instructor, and term
p6 <- ggplot(data = sumthesis1, aes(x = ayear, y = avgscore, size = grades,
                                    shape = semfield, color = semfield)) +
  geom_point() +
  labs(x = "Academic Year", y = "Average # of Attributes", shape = "Field", color = "Field", size = "Assessed")
p6

```


```{r attributes-field}

quality2 <- thesisdata %>% filter(graded == 1) %>%
  group_by(semfield) %>%
  summarise(
            sourceprop = mean(sourcecount)*100,
            conceptprop = mean(concepts)*100,
            litprop = mean(litcritique)*100,
            hypoprop = mean(hypothesis)*100,
            primaryprop = mean(primarydata)*100,
            dataprop = mean(dataviz)*100,
            blackprop = mean(blackpol)*100
            )

knitr::kable(quality2, digits = 3,
             col.names = c("Field", "Sources", "Concepts", "Critique", "Hypothesis", "Primary Source", "Visualization", "Black Polisci"),
             align = 'lcccccccc', padding = 2,
             caption = 'Table 6: The American Politics seminars meet the criteria at a higher level than other fields')

```

Table 6 reinforces the message of Figure 4 by showing the percentage of students who satisfy each criterion by the subfield of the Senior Seminar. The lack of a relationship between the curricular path and capstone outcomes is the product of these huge differences in the approaches of the sections of Senior Seminar. A student's choice of seminar overwhelms the potential benefits or deficits of his prior curricular choices. 

The American politics sections of Senior Seminar provide an opportunity to explore that claim. All thirty-one of the research projects from these sections of Senior Seminar were graded with an identical rubric that assigns scores from 0 to 100. This allows us to make a better comparison of how the curricular path impacts the quality of students' final research projects. Table 7 presents a simple linear regression of students' thesis grades (scored on a 100-point scale) on whether they had taken methods prior to the capstone, the number of upper-level electives taken prior to the capstone, and their cumulative GPA. Consistent with the other results, the curricular path does not have an effect.

```{r regression, results='asis'}

# regression of score
m1 <- lm(score ~ scope + gpa + electives + semfield, data = thesisdata)

# regression of source
m2 <- lm(sources ~ scope + gpa + electives + semfield, data = thesisdata)

# regression of concepts
m3 <- glm(concepts ~ scope + gpa + electives + semfield, data = thesisdata, family = "quasibinomial")

# regression of critique
m4 <- glm(litcritique ~ scope + gpa + electives + semfield, data = thesisdata, family = "quasibinomial")

# regression of hypothesis
m5 <- glm(hypothesis ~ scope + gpa + electives + semfield, data = thesisdata, family = "quasibinomial")

# regression of primary source/data
m6 <- glm(primarydata ~ scope + gpa + electives + semfield, data = thesisdata, family = "quasibinomial")

#regression of dataviz
m7 <- glm(dataviz ~ scope + gpa + electives + semfield, data = thesisdata, family = "quasibinomial")

# regression of thesis grade
m8 <- lm(thesis ~ scope + electives + gpa, data = thesisdata)

stargazer::stargazer(m8, title = "Research grades are driven by GPA and not curriculum.",
                     dep.var.labels = c("Final Project Grade"), 
                     covariate.labels = c("Methods prior to Capstone", "number of Electives", "Cumulative GPA"),
                     header = FALSE, output = 'latex')

```


# The Bigger Picture
Looking at the results of this study of the department's capstone, an easy conclusion would be that curriculum does not matter. Instead, I think these results reveal that we were operating with a flawed definition of "curriculum". I was thinking about curriculum in terms of the sequence of courses outlined in the catalog. The review of course syllabi shows that curriculum has to be thought of in terms of the realities of the classroom. Whichever approach to curricular design a department takes, the faculty need to regularly review how/whether that design is being implemented.Put simply, we cannot expect integrated learning without an integrated curriculum.

The central finding of this study also highlights that the students were coming into Senior Seminar as virtual blank slates. Students did not have a strong or clear sense of what political science research entailed, so they adopted whatever schema their particular section offered. This indicates that the department has not articulated a comprehensive vision of what constitutes a good political science education. The curriculum should take students on a coherent journey with a defined endpoint in mind. Faculty have to buy into that vision in order for any curriculum to function properly. These results reveal distinct notions of "good political science" and potentially of the purpose of Senior Seminar itself. How can we reconcile the idea of a curriculum that articulates a universal vision of political science education with the desire to maintain the existing methodological pluralism? 

Finally, this examination shows that it is possible to successfully provide instruction with a particular focus on mission and a point of view. The student research projects overwhelmingly fit within a broad concept of black political science. Our college has a degree of demographic homogeneity in terms of race and gender that sets it apart from other schools. However, I think there are lessons that can translate into different environments. We do not have departmental meetings in which the meaning of "black political science" is the subject of discussion. Instead, we have been able to achieve this apparent consensus through the hiring process. That is, applicants are explicitly asked to reckon with the college's particular mission, the population of students that we have, and how race is/should be integrated into any understanding of politics. The second component is that students are freely choosing these approaches to their research because that is what is defined as "normal" within our classrooms. We teach political science from the vantage point of an oppressed racial minority, so conventional wisdoms are rarely presented as neutral. Instead, class discussion and lecture push students to examine whether standard theories or approaches apply to black people or regard us as fully human. That ties into the first point above -- curriculum is not what we hope students are doing; it is what actually happens in our classrooms.
\pagebreak


# Bibliography

